Network data stream simulation with time range LDA pattern mining

This project involves the simulation of a SIEM system using Latent Dirichlet Allocation for IoT device streams. It can be implemented in R, Python, C++ or any relevant language that achieves the outcome. Workflow Input config > random & pattern generated content streams > stream chunks > LDA parser > output pattern frequency & topics per stream Data Generation Input config > random & pattern generated content streams The generator should be configurable and able to create network simulation data streams. Each stream generates random content and includes generated content as provided by the config file: 1. stream information 2. string and regex patterns to include in the stream (generator fills the regex with matching values) 3. occurrence frequency (range 0 to 10) which represents the number of the generated string and regex patterns to include per minute. Does not have to be very sophisticated, just relatively different. The generator can be started and stopped. Example inputs configuration for 2 streams in JSON format. / input/[login to view URL] { { “name”: “endpoint1”, “ip”: [login to view URL], “port”: 345, { “pattern”: “IP_EXT: '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}' MSG: ^#[^ !@#$%^&*(),.?":{}|<>]*$ USER: ^[a-z0-9_-]{3,15}$” “frequency”: 2 }, { “pattern”: “PAYLOAD: ^ABC_[^ !@#$%^&*(),.?":{}|<>]*$ ID: ^[a-z0-9_-]{30,150}$” “frequency”: 5 }, }, { “name”: “syslog1”, “ip”: [login to view URL], “port”: 534, { “pattern”: “IP_EXT: '(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}' MSG: ^#[^ !@#$%^&*(),.?":{}|<>]*$ USER: ^[a-z0-9_-]{3,15}$” “frequency”: 2 }, { “pattern”: “PAYLOAD: ^ABC_[^ !@#$%^&*(),.?":{}|<>]*$ ID: ^[a-z0-9_-]{30,150}$” “frequency”: 5 }, }, } Sample stream chunk. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Sed euismod eros a lectus porttitor, vitae aliquet magna ullamcorper. Praesent in enim non magna vehicula faucibus. Vestibulum lacinia velit ut dolor aliquet tincidunt. IP_EXT: [login to view URL] MSG: #abyx USER: das-dkjh Ut consectetur hendrerit massa vel tempus. Nulla sit amet libero id felis lacinia accumsan. PAYLOAD: ABC_aS57dasd USR: 42d8ffe6-8a65-416c-ac92-d5826315faa6 In dictum porta magna sed lectus venenatis. Aliquam accumsan molestie augue, sit lectus amet vulputate metus tristique et. Ut a lectus erat elit…. Regex specifications from [login to view URL] [login to view URL] [login to view URL] Stream Parser stream chunks > LDA parser > output pattern frequency & topics per stream The streams are red by a parser application which reads each input stream for a configurable span of time (e.g. 30 seconds) as input chunks. You must use the Latent Dirichlet Allocation package or method to analyze the data and create/append to 3 log files per stream. Each run is in a new output folder with a timestamp from when the run began. 1. the found matching patterns log (use the input file to identify patterns), 2. the count of the patterns in that timespan log, and 3. up to 10 highest frequency single string terms (LDA topics, occurrence > 1 & not in regex patterns?) Attached is a research paper related to the filed of study. My aim is to replicate the basic stream generation and pattern matching using LDA. It is just a proof of concept and not for production code. Good use of comments is always welcome!