We actually were able to identify what might be the cause of this issue today. It looks like the .github folder is not being created in all students' repos. It is created is some but not all, so when they push, there's nothing actually there to run the tests. When you "edit" the assignment (with no edits) that new commit somehow triggers the .github folder to be created. However, this does not happen consistently so some students still will not have the .github folder in their repo.

Thanks @samus-aran!

I'm currently manually adding the .github folder with the autograding json and classroom yml files to the upstream fork so students can submit their work for this week, but I can replicate this new assignments.

Hey @samus-aran

As of yesterday, the .github folder was still not being created when students initially accept the assignment. I'm creating the assignment from a private repo which has several files including test files. Their forks initially contain all the appropriate files, but not the .github folder. I then have to "Edit" the assignment, make no changes, click update assignment, and the student refreshes their browser and the .github folder is created. I have to do this several times throughout the day, pretty much whenever anyone new accepts the assignment.

Yes, this is happening to my students as well. They don't have a .github folder when they first accept the assignment, but we can get it to appear if I update the assignment with no changes after they accept the assignment.

This workaround works for creating the autograding workflows. However, this way the workflow is not run automatically, as the commit by github-classroom bot doesn't trigger it per the rule in the workflow. So either the students or I have to manually dispatch the workflow in each repository, or it will run when the student pushes commits to it.

There is another problem: the scores are not shown on the assignment page next to the student names. I did the manual dispatch for each but the score is displayed only for 1 student out of 15.

This is still a problem. Not every student repo has a .github directory. I, personally, experienced it today, but have seen it in a lot of the student repos.

I also experience the other problem that @oliverosz . It only detects the executed autograding sometimes, for some students, or perhaps it just happens very seldom.

These two problems make it impossible to use the autograding feature as intended. Which is sad, as I think it's nice. Could you or someone please take a look @samus-aran ?

Also, The "Enable feedback pull requests" option is not working anymore.

Thank you @lmraine for your commitment sharing your experiences! We've passed this onto the internal team to investigate.

@mcknijak , @pinventado, @joaquinvanschoren, @locbaoodo Are you also experiencing some discrepancies since the latest update (February 29th)?

Hi @samus-aran . I'm still experiencing this issue. A month ago I created an assignment and throughout this month the .github folder was not created automatically in about 10% of the students' repos.

Yesterday I created three new assignments for an exam we will do today and joined myself to test if they were working correctly. The .github folder was not created in my repo when I joined the first one. I followed @lmraine workaround (edit assignment, make no edits, save the assignment) and got the .github folder. Then, I joined the second and third assignments but in those cases the folder was properly created when I joined.

@aaleta & @lmraine are you able to share with us the names of your classrooms and the assignments that are still problematic? Alternatively you can open a GitHub Education support ticket (select GitHub Classroom as your option) with this info and share the ref here so I can highlight it to the team 👍

So yesterday three groups of students took the assignments, and I kept track of the issues. The assignments are private so let me know if you need more information to access them, or if it's better to submit a support ticket.

The first assignment was:

https://classroom.github.com/classrooms/149118562-computacional-unizar/assignments/prueba-1a-el-oscilador

A total of 25 students joined between 10:00-10:10 (CET) (+me, 26). The .github folder was not created automatically by the bot in 12 of them. As I mentioned, I also didn't get the .github folder when I joined myself.

The second was:

https://classroom.github.com/classrooms/149118562-computacional-unizar/assignments/prueba-1b-el-oscilador

29 students joined between 15:05-15:15 (CET) (+me, 30). 7 of them did not get the .github folder.

The third one was:

https://classroom.github.com/classrooms/149118562-computacional-unizar/assignments/prueba-1c-el-oscilador

19 students joined between 17:15-17:25(CET) (+me, 20). 9 of them did not get the .github folder.

In all three cases, I opened the edit button, saved the assignment without making any changes, and the .github folder was automatically added to all the repos that didn't have it. The only issue is that then some students had to do a pull to sync their local repos before pushing their commits, but other than that there were no further issues.

Thanks @aaleta I've shared this with the team. 👍

Now the workflows work properly if I supply the .github folder in the teacher repository. It gets copied into the students' forks and workflows are enabled. 👏 However, the "add deadline" commit was not made by classroom-bot in some of the repos.

I actually prefer this method to the online wizard for defining tests, as it's easier to copy test commands in the autograding.json than filling out separate forms for each test. The latter method returned an internal server error (500) when I tried to save the new assignment, and the tests were lost.

Thanks for your feedback @oliverosz 🙇

However, the "add deadline" commit was not made by classroom-bot in some of the repos.

But it was expected? I'll pass this over to the internal team to review. 👍

The latter method returned an internal server error (500) when I tried to save the new assignment, and the tests were lost.

Are you able to provide steps to reproduce this? What actions did you take?

However, the "add deadline" commit was not made by classroom-bot in some of the repos.

But it was expected? I'll pass this over to the internal team to review. 👍

I think it's the same problem as the .github folder not being created, as it is normally created by classroom-bot in another commit when autograding tests are added by the wizard, not in the .github folder of the source repository.

The latter method returned an internal server error (500) when I tried to save the new assignment, and the tests were lost.

Are you able to provide steps to reproduce this? What actions did you take?

I played around with it to see when it occurs. I only get an error if I add a test case with test name 1 at the Grading and feedback page. Weird. 😃 I left all other settings on default, tried with other settings and different test types as well, it's consistent: the test case name cannot be 1. If editing an already existing assignment, there is no problem with name "1" and it gets saved.

Hi @lmraine 👋

@aaleta & @lmraine are you able to share with us the names of your classrooms and the assignments that are still problematic? Alternatively you can open a GitHub Education support ticket (select GitHub Classroom as your option) with this info and share the ref here so I can highlight it to the team 👍

Sadly I tried your fix, but it did not work for me.

This in an internal problem of github because I cannot duplicate the test.

@luis-damas-isel thanks for bringing this to our attention. Looks like we have a bug with our input parsing logic that needs to be fixed.

@luis-damas-isel we just merged in a fix for this issue, could you try again?

Hi.
Now is working fine.
The only thing not working is that I cannot "Reuse assignment" witch always fails.

Going to echo @LuisDamas report here, the "Reuse assignment" functionality appears to fail silently. I do not see any error given and the assignment is simply not created in the target classroom

@ashishkeshan Thanks! Could you please also check what's wrong here?
https://github.com/ML-course/assignment-3-joaquinvanschoren-3

All I get is 'FAILED submission_test.py::test_submission - Failed: Error while running notebook.'
but when I run it locally in a clean venv it runs fine.

Looks like there's a bug with our autograding-python-grader. We will roll back the code. In the mean time, could you change this:

 - name: submission_test
      id: submission_test
      uses: education/autograding-python-grader@v1
      with:
        timeout: 60

to be:

- name: submission_test
  id: submission_test
  uses: education/autograding-command-grader@v1
  with:
    test-name: submission_test
    setup-command: sudo -H pip3 install --upgrade pip; sudo -H pip3 install setuptools; sudo -H pip3 install -U -r requirements.txt
    command: pytest
    timeout: 60

@joaquinvanschoren did that fix your issue?

@ashishkeshan Yes, thanks! Can this fix be deployed to all other repositories for this assignment?

If you edit the assignment test to be a run_command test with those parameters in the UI, it should update the tests in every student repo.

What's the error you're getting?

@nineil-weber will take a look at our parsing logic to see why that character is getting added. Thanks for bringing it to our attention.

@nineil-weber we just merged in a fix for this issue, could you try again?

HI @ashishkeshan, The same problem still continues.

You'll need to recreate the assignment for the changes to take place

hI @ashishkeshan , I just created the assignment again and everything is working correctly. Thank you so much.

Sorry, I spoke too soon here, it looks like entire workflow process is being reworked as the new .github/workflows/classroom.yml file appears to have the content of the .github/classroom/autograder.json file. Notably absent from steps in the workflow is - uses: education/autograding@v1 so it seems like we're trying to phase out using that and the two separate files approach? The workaround will have to be copying both workflows and classroom folders from known working repos until this is sorted I guess... ☹️

Just want to finally confirm that moving folders from repos with known working autograder workflows does get around this issue.

Also now discovering that making an assignment inactive (since it for some reason invokes the bot on everyone's repo) screws up the autograder for every repo under that assignment. So thanks a lot for that... 😞

Going to just update my specific thread to note that a student has discovered that test cases that are named starting with a number appear to cause this specific failure. I am not going to modify test cases now as we are at end of semester, but will try to confirm this later on this month...

The test results from pytest should be showing up, could you link me to the assignment you've created so I can check it out to debug with?

Doubtful you will receive a reply here, but classroom team screwing with autograder gh actions generation mid-semester is definitely a big mistake. Kind of baffling considering they took the right steps when it came to switching repos over to forking model instead of template repos (it was opt in this semester and won't go live until June 17th).

Just putting this here FYI: it seems you must not name test cases starting with a number as one of my students has discovered. This is what is causing mine to fail. It seems like you have a different issue and your cases are not named this way. Sorry you have 126 test cases to update :(

Perhaps you can script your browser to do this work? They have not exposed any API for dealing with test cases and the given UI is very clunky still...

Done! :-)

@luis-damas-isel which repo should I look at?

Classrooms | estsetubal-bd-2023-24-classroom-ctesp-amadora | 02-select-groupy-by-Template
user with problematic ratings ...658@.....

@luis-damas-isel I just checked it out and looks like clasroom is pulling the info from an older commit that received a score of 19/20. I think it didn't update with the later commits because it thinks they were submitted late. Did you have a deadline for the assignment?

The original commit that was graded with 20/20 in the details and appear as 19/20 in the assignment details.
Than I forced a change in a file to force to re-run the autograder to see if the correct value appear (which is marked as "Late") but it remain with 19/20, also if you download the scores.