Benchmarking SolidWorks.

I am trying to put all benchmarks in one place. If you know of anything
else that should be there let me know.

http://www.engtran.com/SWBM001.html

There is a wish list for benchmarks. Except for SPECapc none of the
others do anything with assemblies or drawings. In addition to
automated assemblies there has to be a way to assess the user interface
which would require user interaction.

 

I am asking the question: How do you benchmark a user interface? This
is much more difficult than running a script because it requires the
user to do something and this introduces a certain amount of randomness
to the test. Some things in the UI can be benchmarked I believe. Some
have complained that the UI seems slower or more sluggish. It might be
possible to capture that.

I used to watch a certain TTM run through a SW demo and got to thinking
that he was doing it about as fast as humanly possible. I have never
seen a normal user construct parts and assemblies that fast. So what he
was doing was the limit on the interface.

I also consider the SW challenge at SWW a form of benchmark. There are
many variables. It is totally up to the user to determine the solution
to the problem and then to implement it. The fact that so few finish,
but that some finish in a very short time gives some feel for SW
performance.

 

I have seen macros written in the application that track how far the
mouse moves or number of clicks, there is a really cool one I have seen
called mouse miles or something like that that just tracks how many
miles you travel with your mouse after a while.

This would give you an indication of user interface. Turn such a macro
on and give the user a few tasks to do. Get the above information and
maybe even an over all time. This would only really be useful to test
various users or even different versions of the software as they move
things around or create compound functions.

--Matt

 

Now that this link has been out a while I would really like to see some
results, especially for high and middle Intel stuff. I just have AMD
myself so can't really compare unless it be with a 650Mhz laptop.

 

I think it is a little more complex than mouse miles, although that
sounds pretty cool to see. The ease of an interface could be measured
in number of steps, keyboard actions, and even mouse mileage. However,
there are also the less easily quantified aspects of logical layout,
intuitive placement of icons or menues, how large clickable zones are,
that enhance workflow and the user experience (note the debate between
Windows and Mac OS which has a lot to do with these less easily
quantified user experiences).

For example, if you are familiar with Alias StudioTools / Maya marking
menu system, you would probably agree that it is a nightmare for the
new or casual user, but jockeys that "know" it and use it daily find it
significantly faster than using a normal menu system. But on a casual
basis I really find it frustrating. When I have a job where I use it
for a few days, I enjoy it again... then forget it again.

Now if mouse milage could be converted to calories burned, we could
also see which CAD program has better health benefits ;-)

Daniel

 

I totally agree, I was just trying to give one example. I for one
actually like pro/e's marching menus as I could get around fast with
those. I am also one that likes quick keys and couldn't get along
without them. In terms of the interface one of the huge benefits is the
ability to use quick keys and even api. I find as soon as I do things a
few times, I script them and tie them to quick keys. How easy is this
to do today? But then how easy is it to transfer this to another user,
sometimes difficult.

I even know people whom have learned to use the mouse with the left hand
as it makes so much more sense in the CAD world to have your right hand
on the num-pad. I haven't been able to train myself that way....

But with that said #of clicks, mileage and average velocity for a set
number of functions do actually test much of what you describe below.
And the goal of most benchmarks is a quick test to show relative
numbers. The problem is trying to level the playing field in a way
these numbers actually mean something outside of the test.

--Matt

 

The other problem with this metric is actually making the comparison.
But then this tends to lead back to the question of what is to be gained
by this benchmark. The benchmarks I tend to do has an outcome,
replacement or justification of hardware etc. And that tends to lead
toward how I do the benchmark. We are currently looking into gig-a-bit
to the desktop, as it is the next step as we just moved over to SAN
technology for our network storage and have really gotten storage as
fast as we can.

--Matt