Post by account_disabled on Feb 20, 2018 4:52:11 GMT
Hi,
First of all, thank you for the great work in this library that I discovered few weeks ago. I am back to NN after more than 10 years, and I am glad such "tools" are now public.
Some questions about network design and code implementation:
I have massive amount of data, normalized (beetween 0 - 1), and a single output normalized as well (-1 to 1).
So far I can set up to 58 variables in input.
I can get up to 2.5 millions observations.
58 input is big, and I do not know their relevance in the prediction, so I am very glad to use the cascade_training. However, millions of observations will take ages, so I decided to use several smaller sample randomly.
The amount of data allows me to create several training data set and test data set. I did play around with the cascading and I actually have over fitting problems (ie: training error 0.000099, testing 0.028222).
The solution that came up to me is to use the callback during the cascade training to run the network on several test data file.
I have a problem here, due to my lack of understanding of the FANN underlying.
When the training is running, it seems that it would need to keep the current state/error/other values (for rprop etc..) to continue the training. So if I use the same network to test some data (in the callback), and decide that the training can keep going (regarding the test results error), do I need to restore some previous internal state of the network (means I would have to "Save" and internal state prior to run the tests) ?
Please help.
Thanks!
I didn't find the right solution from the Internet.
References:
leensdgdfissen.dk/fann//forum/viewtopic.php?f=1&t=492
Microfinance Examples
First of all, thank you for the great work in this library that I discovered few weeks ago. I am back to NN after more than 10 years, and I am glad such "tools" are now public.
Some questions about network design and code implementation:
I have massive amount of data, normalized (beetween 0 - 1), and a single output normalized as well (-1 to 1).
So far I can set up to 58 variables in input.
I can get up to 2.5 millions observations.
58 input is big, and I do not know their relevance in the prediction, so I am very glad to use the cascade_training. However, millions of observations will take ages, so I decided to use several smaller sample randomly.
The amount of data allows me to create several training data set and test data set. I did play around with the cascading and I actually have over fitting problems (ie: training error 0.000099, testing 0.028222).
The solution that came up to me is to use the callback during the cascade training to run the network on several test data file.
I have a problem here, due to my lack of understanding of the FANN underlying.
When the training is running, it seems that it would need to keep the current state/error/other values (for rprop etc..) to continue the training. So if I use the same network to test some data (in the callback), and decide that the training can keep going (regarding the test results error), do I need to restore some previous internal state of the network (means I would have to "Save" and internal state prior to run the tests) ?
Please help.
Thanks!
I didn't find the right solution from the Internet.
References:
leensdgdfissen.dk/fann//forum/viewtopic.php?f=1&t=492
Microfinance Examples