#40 Bytecode-based "timing" limits for python code

Closed
opened 2 months ago by hz · 1 comments
hz commented 2 months ago

It would be great to have a more reliable way to use timing to measure efficiency of student solutions (as opposed to just using it to check for infinite loops). Using a count of the number of opcodes executed during a particular run, while not perfect, would be likely to give us more finely-grained control over efficiency for scoring purposes.

I think this is very doable. I remembered this being hard in the past, but I was looking around the CPython source today for a way to hack this in, and it turns out that it is already available (new in version 3.7), via the usual sys.settrace function. Here is a minimal example that could be integrated into one or more of the Python sandboxes:

import sys

def trace_closure(verbose=False, limit=float('inf')):
    executed_opcodes = 0

    def tracer(frame, event, arg):
        nonlocal executed_opcodes
        frame.f_trace_opcodes = True
        if event == 'opcode':
            executed_opcodes += 1
            if executed_opcodes > limit:
                sys.exit("tracer: opcode limit reached")
            if verbose:
                print(frame, event, arg)
        return tracer

    def get():
        return executed_opcodes

    names = {
        'tracer': tracer,
        'get': get,
    }

    return lambda n: names[n]

tracer = trace_closure(verbose=True, limit=100000)
sys.settrace(tracer('tracer'))

import sft

sys.settrace(None)
print('_opcodes_executed=%d' % tracer('get')())
It would be great to have a more reliable way to use timing to measure efficiency of student solutions (as opposed to just using it to check for infinite loops). Using a count of the number of opcodes executed during a particular run, while not perfect, would be likely to give us more finely-grained control over efficiency for scoring purposes. I think this is very doable. I remembered this being hard in the past, but I was looking around the CPython source today for a way to hack this in, and it turns out that it is already available (new in version 3.7), via the usual [`sys.settrace`](https://docs.python.org/3/library/sys.html#sys.settrace) function. Here is a minimal example that could be integrated into one or more of the Python sandboxes: ```python import sys def trace_closure(verbose=False, limit=float('inf')): executed_opcodes = 0 def tracer(frame, event, arg): nonlocal executed_opcodes frame.f_trace_opcodes = True if event == 'opcode': executed_opcodes += 1 if executed_opcodes > limit: sys.exit("tracer: opcode limit reached") if verbose: print(frame, event, arg) return tracer def get(): return executed_opcodes names = { 'tracer': tracer, 'get': get, } return lambda n: names[n] tracer = trace_closure(verbose=True, limit=100000) sys.settrace(tracer('tracer')) import sft sys.settrace(None) print('_opcodes_executed=%d' % tracer('get')()) ```
hz added this to the 14.0.0 milestone 2 months ago
hz added the
question type
label 2 months ago
hz added the
enhancement
label 2 months ago
hz self-assigned this 2 months ago
hz commented 2 months ago
Owner

Implemented in #46

Implemented in #46
Sign in to join this conversation.
No Milestone
No Assignees
1 Participants
Due Date

No due date set.

Dependencies

This issue currently doesn't have any dependencies.

Loading…
Cancel
Save
There is no content yet.