BIGDATA SIMULATION LIBRARY OF DREAM BY SANDMAN BOOKS
=================
Solution Architecture
Description
In the realm of Dreaming, its ruler SANDMAN, DREAM has a certain hobby; books. In his castle there is a Library in which they are kept, among other things, stories conceived by their authors but never written in our reality; Lucien, the person responsible for his organization, needs some help. Many people dream of published books, sales markets, stories that, in their reality, they would never imagine conceiving. And this voluminous data needs to be worked on. In order not to get lost in the information, Lucien receives all his dreams in a Non-Relational bank, MONGO. And he needs this to be organized in a relational way, that is, each author in his proper place. For that he pulled our dream and saw this Architecture where data arrives in MONGO undergo a transformation process in the STAGIN area and are populated in MYSQL. In its population, we split two final tables. One in its raw state, for complete queries, and another with metrics that informs the number of dreamers, their books and the total number of files. In this way, data is more organized, undergoing deduplication and consolidation processes.
Glossary of Data
| Fields | Type | Description |
|---|---|---|
| _id | long | undescore ID |
| kind | string | type of book or text file |
| title | string | title of book |
| subtitle | string | subtitle of book |
| author | array | one or more authors who can dream of stories |
| publisher | string | publisher or not dreamed of by the author |
| publishedDate | string | year of published |
| edition | string | which edition does the book belong to |
| sample | string | sample of books |
| type | string | ISBN code |
| identifier | string | isbn identification number |
| pageCount | integer | number of pages |
| capCount | integer | number of chapters |
| wordCount | integer | number of words |
| categories | string | literary genre |
| original_price | double | original price |
| current_prefix | string | country currency prefix |
| current_sufix | string | country currency name |
| barcode | string | barcode |
| dreaming_date | string | the day you had the dream |
image
Start the Project
To run the project, you need to install the dependencies located in the "dependencies" folder and in the root of the project, run the shell_script "run_script.sh".
Sample of Payload MONGO
mongo
{
"_id" : ObjectId("61b1fe6944dd42158674af31"),
"kind" : "books#volume",
"volumeInfo" : {
"title" : "STORE VISIT",
"Subtitle" : "GLASS IT HAIR MEMBER KEY ALMOST QUALITY. MARKET ALREADY AIR STILL ARTICLE. DECADE DECADE MEASURE PRESENT HUMAN MORNING. BIG BLOOD ECONOMIC FRONT SUCCESS AGO THEM. EVERY SON TROUBLE SIMPLE.",
"author" : [
"PETER RODRIGUEZ",
"KELLY TORRES"
]
},
"publisher" : "FALL AWAY ABOUT INDEPENDENT",
"publishedDate" : "1994",
"edition" : "7º EDITION",
"sample" : "...onto sport room audience. page dinner hundred. week statement should watch she even ball.\nour able tv break defense seek baby. employee last around music produce reach tv..",
"industryIdentifiers" : [
{
"type" : "ISBN_10",
"identifier" : "1-55027-208-X"
},
{
"type" : "ISBN_10",
"identifier" : "0-405-30324-6"
}
],
"pageCount" : 796,
"wordCount" : 83331,
"capCount" : 14,
"categories" : [
"NOVEL"
],
"saleInfo" : {
"original_price" : 78,
"current_prefix" : "LAK",
"current_sufix" : "Lao kip",
"barcode" : "6747254889534"
}
}
Sample of Payload in MYSQL
library
_id |kind |title |subtitle |author |publisher |publishedDate|edition |sample |type |identifier |pageCount|wordCount|capCount|categories |original_price|current_prefix|current_sufix |barcode |dreaming_date|
-----+------------+----------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+---------------------------------------------+----------------------------------------+-------------+----------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------+-----------------+---------+---------+--------+-------------------------+--------------+--------------+---------------------------+-------------+-------------+
670|csv#volume |GOOD DETERMINE OF |DON'T HAVE |KAREN ODOM |DARK WAR INDEPENDENT |1982 |9º EDITION|...involve star apply later including truth. next while nor worry staff economic.¶condition region write college. return half offer. popular could direction above fish.. |ISBN_10|978-1-4340-7508-6| 479| 64333| 11|EPISTOLARY NOVEL | 97.0|BHD |Bahraini dinar |3894426059691| 20211209|
resume
metric |value|
----------------+-----+
Total of Dreamns|71395|
Total of Books |59154|
Total of Data |78000|
1.8k Jan 09, 2023
41 Jun 16, 2022
2.5k Dec 30, 2022
2 Dec 13, 2022
88 Nov 04, 2022
5 Jul 13, 2022
3.7k Jan 03, 2023
529 Dec 19, 2022
5 Mar 19, 2022
8.2k Jan 07, 2023
0 Jan 20, 2022
369 Dec 30, 2022
18 Nov 28, 2022
1 Jul 27, 2021
1 Nov 01, 2021
61 Nov 21, 2022
2 Dec 22, 2021
2.5k Jan 09, 2023
14 Jan 02, 2023
102 Nov 16, 2022