Artifact Evaluation in Hardware Research


"For the straightforward pathway had been lost."
— "Dante's Inferno"


Hardware research has a crisis. I would say it's a reproducibility crisis, but it's worse than that. In a sense, it's a core issue deep in the field and in the community. I don't know if I can do anything about it, but it is always good to talk about issues on your mind like all good therapy. Maybe I'm not alone.

Artifact Evaluation

To talk about the issue I want to talk about, I will pick on the idea of artifact evaluation for the time being.

Artifact evaluation is an additional step in the conference or journal review process that allows authors to submit code, data, binaries, hardware designs, hardware access, and other digital "objects" for reviews to reproduce and validate results in the submission. The most common framework of artifact evaluation in my areas of research is one by the Association for Computing Machinery (ACM), as it covers a wide range of software and hardware conferences and journals \cite{noauthor_artifact_2020}. However, artifact evaluation is also present in conferences and journals outside the scope of ACM (such as IEEE or big ML conferences).

Artifact evaluation is also not necessarily a centralized requirement; venues typically adopt their own artifact evaluation system or use an already proposed system like the one from the ACM. Regardless, many software and hardware conferences have adopted artifact evaluation in their review process.

The final and most important detail is that artifact evaluation in hardware research fields is almost always optional. It is separate from the normal review process and is not required for publication. If your paper is accepted, you can still publish it without submitting supplemental materials for artifact evaluation.

Stop complaining and be productive...

Many people in academia may see this whole thing as me just complaining. Complaining that "it's not fair". Complaining about how hard it is for me to get accepted submission. Complaining when things already work the way they are and have been fine for a while. Complaining about things that are not productive to complain about and wasting everyone's time. Complaining that my research is just not good enough and I'm using this as a crutch.

I agree. But the critics are still wrong. Sure I'm a grad student with an idealistic outlook on academia and that we can fix the things in the field that are broken. And if I offer solutions to the problems, and actively work towards them, then in my heart, I can rest easy.

I understand that things have been chugging along for a while without issue or fail. However, that's certainly a lie. Many issues have been ignored or swept under the rug. And now we are at a point where we can't ignore them anymore.

If people think that this crusade will get me nowhere in an academic career, then we are all doomed. If I can't be a successful academic, point out the problems in the field, and then try and fix them, then what is the point of being an academic? And if I try and I fail, I will live a fulfilled life.