I am working on a project that takes information from hardware and stores it into a DB every 20ms. While it works 100%, the screen only refreshes about every 5s.

My question is, will switching to flat files speed up processing time and give me a faster screen refresh time?

With the sql I am able to grab and send information easily. Writing the code for handling flat files is a non-issue, however I do not want to begin re-writing this project until I am certain that it will inhance the programs performance (mabey even be able to speed screen refresh rate up to close to 20ms).

Thanks,
Sean

: I am working on a project that takes information from hardware and stores it into a DB every 20ms. While it works 100%, the screen only refreshes about every 5s.
:
: My question is, will switching to flat files speed up processing time and give me a faster screen refresh time?
:
: With the sql I am able to grab and send information easily. Writing the code for handling flat files is a non-issue, however I do not want to begin re-writing this project until I am certain that it will inhance the programs performance (mabey even be able to speed screen refresh rate up to close to 20ms).
:
: Thanks,
: Sean

Put a DoEvents call in your main loop, and it will probably be enough.

: Put a DoEvents call in your main loop, and it will probably be enough.

Already using DoEvents.

Thanks for your response,
Sean

: I am working on a project that takes information from hardware and stores it into a DB every 20ms. While it works 100%, the screen only refreshes about every 5s.
:
: My question is, will switching to flat files speed up processing time and give me a faster screen refresh time?
:
: With the sql I am able to grab and send information easily. Writing the code for handling flat files is a non-issue, however I do not want to begin re-writing this project until I am certain that it will inhance the programs performance (mabey even be able to speed screen refresh rate up to close to 20ms).

I think using a flat file would probably be faster, but only to a certain point. And it would depend on if you're just constantly appending to an open file (that stays open the whole time), or if you're repeatedly opening/appending/closing the file. It'd be even worse if you had to parse and re-write the file every time. Plus, you lose any flexibility you may have with an SQL engine as your store.

Like Genjuro said, the DoEvents statement might help.

Another thing you might look into is a library called SQLite (for which there are VB libraries available, though I've never used it myself) which essentially gives your program an in-memory database engine instead of communicating with a separate database process.


infidel

$ select * from users where clue > 0
no rows returned

: I think using a flat file would probably be faster, but only to a certain point. And it would depend on if you're just constantly appending to an open file (that stays open the whole time), or if you're repeatedly opening/appending/closing the file. It'd be even worse if you had to parse and re-write the file every time. Plus, you lose any flexibility you may have with an SQL engine as your store.

: Like Genjuro said, the DoEvents statement might help.

: Another thing you might look into is a library called SQLite (for which there are VB libraries available, though I've never used it myself) which essentially gives your program an in-memory database engine instead of communicating with a separate database process.
------------

I am already using DoEvents.

I don't think we would have to re-write the file every time if it's open randomly right?

Thanks for your response,
Sean

: I don't think we would have to re-write the file every time if it's open randomly right?

No, you don't. "Random" means you can open it and insert or read data from wherever you want to without having to rewrite the whole thing. If you need some sample code I've got some. There's plenty out on the net if you do a search for it as well.

: No, you don't. "Random" means you can open it and insert or read data from wherever you want to without having to rewrite the whole thing. If you need some sample code I've got some. There's plenty out on the net if you do a search for it as well.

I thought so. I do not need sample code, I already know how to do this in other languages, but am somewhat new to VB.

Thanks for your response,
Sean

: I am working on a project that takes information from hardware and stores it into a DB every 20ms. While it works 100%, the screen only refreshes about every 5s.
:
: My question is, will switching to flat files speed up processing time and give me a faster screen refresh time?
:
: With the sql I am able to grab and send information easily. Writing the code for handling flat files is a non-issue, however I do not want to begin re-writing this project until I am certain that it will inhance the programs performance (mabey even be able to speed screen refresh rate up to close to 20ms).

Explain a bit more about the screen refreshing. Are you pulling the data from the DB onto a form? Are you using ADO to do this? Are you databinding controls?
Were it me I'd stick with using SQL, because unless it's LARGE amounts of data (lots of characters... and I mean lots!) then SQL should be as fast as writing to a file. Are you writing this data to a database on the network, or is it local to your machine? I'd try to figure out where the bottleneck is occuring first before changing something as major as how you read and write the data, but that's just me. ;)

This message was edited by seancampbell at 2005-6-24 10:10:7

---

: Explain a bit more about the screen refreshing. Are you pulling the
: data from the DB onto a form? Are you using ADO to do this? Are you
: databinding controls?

Information is pulled from hardware into variables which are then using those to update the form and store the information in a Database.

As far as connecting to the Database goes, Its using ADO, not datacontrols.

: Were it me I'd stick with using SQL, because unless it's LARGE
: amounts of data (lots of characters... and I mean lots!) then SQL
: should be as fast as writing to a file. Are you writing this data to
: a database on the network, or is it local to your machine? I'd try to
: figure out where the bottleneck is occuring first before changing
: something as major as how you read and write the data, but that's
: just me. ;)

Well the project is already completed, I am documenting the code for my boss. It is my job to learn all of the ins and outs of the project so I can rebuild it with flat files.

I am not aloud to talk to in depth about the project, however I will say what I can.

The program is taking information received from levers buttons and sensors and storing it into a database. This information is added every 20 milliseconds and the program usually runs for atleast 24 hours.

The problem is that the amount of information being stored into the databases is slowing down the refreshing rate on the forms. So although the information is being stored in real-time, I am only able to see the changes about once every five seconds or more (I actually have only seen the program running twice).

Now my bosses seem to think that changing over to flat files will solve their problem, however I am concerned that I may spend 9 months working on a project that is not improving (or worse, making it worse) the performance of the program.

I hope this is enough for some further help.

Thanks again for your response,
Sean

You're smart for researching the project if it's going to take you that long to complete it and *maybe* still not even yield desirable results.
One thing you can do is try to test it on a smaller scale to see if it may work. The only problem with this is that smaller scale typically means smaller amounts of data, and in turn the results can be skewed from it.
I just have a real hard time believing that writing to a file is going to speed it up immensely. Does your boss have any reasons for this belief?
If you really are writing data every 20 milliseconds for a period of 24 hours (that's a lot disk i/o) then I would maybe look at the system specs. Is a sata harddrive going to speed it up at all? What about a 10000 rpm harddrive? I'm just pulling these ideas out of my head as they come, so if they sound silly that's why. :)
Also, maybe this is just my lack of understanding, but if you are writing data every 20 ms and also trying to update the screen every second at the same time it really doesn't suprise me that you're noticing a bit of latency. If the cpu is that busy then maybe you could look at not writing data for a second or two while you try to read from the database to update the screen. Do you follow me?

This message was edited by seancampbell at 2005-6-24 11:44:21

---

This message was edited by seancampbell at 2005-6-24 11:37:59

---

: You're smart for researching the project if it's going to take you
: that long to complete it and *maybe* still not even yield desirable
: results.
: One thing you can do is try to test it on a smaller scale to see if
: it may work. The only problem with this is that smaller scale
: typically means smaller amounts of data, and in turn the results can
: be skewed from it.
: I just have a real hard time believing that writing to a file is
: going to speed it up immensely. Does your boss have any reasons for
: this belief?
: If you really are writing data every 20 milliseconds for a period of
: 24 hours (that's a lot disk i/o) then I would maybe look at the
: system specs. Is a sata harddrive going to speed it up at all? What
: about a 10000 rpm harddrive? I'm just pulling these ideas out of my
: head as they come, so if they sound silly that's why. :)
: Also, maybe this is just my lack of understanding, but if you are
: writing data every 20 ms and also trying to update the screen every
: second at the same time it really doesn't suprise me that you're
: noticing a bit of latency. If the cpu is that busy then maybe you
: could look at not writing data for a second or two while you try to
: read from the database to update the screen. Do you follow me?

Yeah. I see what you are saying. Well, the problem with refreshing the screen is even at the begining of the run it is running slow. I am certain that these hardware tweaks may change it, but the program is not part of a system. Its actually something you install onto a computer with a PCI card installed.

Thank you for your help, I think that your ideas have steered me in the correct path I need for more research on the subject. I have a few months before I finish documenting the current build, so I have time to look into this.

Sean

!---

After I wrote this, I talked to my boss and he told me that we build the box that we send with the hardware. I am going to write a program that does a stress test on using DBs as apposed to Flat files. I figure this is going to take a few days which meens more work for me.

So as far as file space goes, we don't have a problem there. Also, remember that information stored in a text file is smaller than that of a DB and text files compress quite nicely.





Thanks plenty for your help,
Sean

oh, forgot to mention, is the database optimized (ie - indexes, foreign keys, etc.)?

: oh, forgot to mention, is the database optimized (ie - indexes,
: foreign keys, etc.)?

Yup, optimized like a mad man.

Thanks for your response,
Sean

: I am working on a project that takes information from hardware and stores it into a DB every 20ms. While it works 100%, the screen only refreshes about every 5s.
:
: My question is, will switching to flat files speed up processing time and give me a faster screen refresh time?
:
: With the sql I am able to grab and send information easily. Writing the code for handling flat files is a non-issue, however I do not want to begin re-writing this project until I am certain that it will inhance the programs performance (mabey even be able to speed screen refresh rate up to close to 20ms).

Have you considered splitting it into two programs? Perhaps having separate processes running might help. One process could gather the data and display it but instead of doing the database calls, could use a pipe or some other queueing technique to push the data to a separate process, which just funnels the data into the database. It's just a guess, but perhaps if the GUI part wasn't concerning itself with the storage of the data it would achieve better performance.


infidel

$ select * from users where clue > 0
no rows returned

This message was edited by seancampbell at 2005-6-24 12:24:40

---

: Have you considered splitting it into two programs? Perhaps having
: separate processes running might help. One process could gather the
: data and display it but instead of doing the database calls, could
: use a pipe or some other queueing technique to push the data to a
: separate process, which just funnels the data into the database.
: It's just a guess, but perhaps if the GUI part wasn't concerning
: itself with the storage of the data it would achieve better
: performance.

That actually is a very good idea. I do not know how the current build would be able to incorporate that, but that may be a much better alternitive to rewriting the program using a different file format.

Thank you again for your response,
Sean

-Atleast you didn't mock me about doing homework assignments ;)

: That actually is a very good idea.

What do you mean by "actually" :-P

: -Atleast you didn't mock me about doing homework assignments ;)

Certain inquiries just have a homeworkish smell to them. Yours didn't.


infidel

$ select * from users where clue > 0
no rows returned

: : -Atleast you didn't mock me about doing homework assignments ;)
:
: Certain inquiries just have a homeworkish smell to them. Yours didn't.

I'll agree to that.
Lots of times if they don't require any thought process then it's likely to be a homework assignment.
...
hmm....
What does that tell us about university students these days? :)

This message was edited by seancampbell at 2005-6-25 6:45:0

---

: I'll agree to that.
: Lots of times if they don't require any thought process then it's
: likely to be a homework assignment.
: ...
: hmm....
: What does that tell us about university students these days? :)

Come on now. I am 19, and only a year into college!

However I managed to land a job programming alreay :D