Another peculiar preoccupation you have is with speed. It is the primary cause of your perpetual ruminating. IMO it boils down to what kind of orders do you think your system is going to be using. This is something I don't think you've expressed in your videos. From your speed obsession I imagine you think you are going to be blasting out tons of markets orders. This is probably an ill-advised strategy. If you could have a limit order based strategy you would not need to be faster than the professionals. If you really want to take the speed route with market orders, can you expect to get filled how you would like? Also in the off chance your strategy really does work well, wouldn't it also likely be seen as toxic flow from the perspective of your broker?
You have a knack for making everything oddly complicated. I really don't see why the DB question is important. You don't even need a DB, just save the data in files in folders. You will not be doing queries which necessitate a DB, just retrieving data. The DB or data files are for storage and retrieval only, the data in your program's RAM is where the work is done. You seem to think your program should be interacting with the DB heavily beyond just loading a big batch of data in the beginning and saving any additional data in the end. Even more crazy is saving a tick of data per row in a DB. Who cares if the DB can do a million ticks a second, it's nuts.
Hi Bryan, I am kind of confused after watching. I am in Java for final production and that's all i need. Brokers give API's for charts, live/historical feeds, indicators , even DB connectivity, or CSV through Maven repositories and java platform they provide. Why do i need to store ticks in DB live? I am currently downloading tick data+slippage+spreads in one file, from dukas and work on it for prototyping. I want to start using Matlab for machine learning, i know its easiest way to start there, than ill move to encog or python libraries. But still why Erlang in between and DBs for live ticks…I just log in to brokers servers for data.