Command line tool for monitoring changes of File entities scoped in a Synapse File View

Fixes #24 Fixes #42

• Monitors a single file entity
• Removes File from not implemented list in tests
• Add Schema in addition to File as valid single entity types for monitoring

• Creates synapsemonitor monitor cli command
• Monitors a single Synapse project/entities scoped in a fileview for changes. Will notify users for any files that were added or updated in the past X number of days.

Motivate users to use file views if there is a desire to track files from many projects.

Fixes #25

Allow user to specify value of time in the past to search for modified entities in days, hours, or minutes:

major changes

• adapt CLI to specify value and unit (default is still 1 day)
• adapt function parameters for value and unit input
• adapt detection of modified entities in a fileview and from the entity modifiedOn timestamp to time unit
• new tests for detecting changes by hour and minute

other changes

• use datetime.utcnow() to get current time in UTC directly

Fixes #23

• added _traverse: traversal function that returns any descendants of a given entity type (retained to enhance later functionality)
• implemented _find_modified_entities_container: checks container and descendants for modification
• removed test for unimplemented functions
• added tests in TestModifiedContainer class

Is this worth retaining and maintaining if single entities and containers can be monitored? For efficiency in monitoring a large container's contents, synapsemonitor could create a temporary project and file view under the hood instead of making the user to create one in a separate command.

1. Create (if doesn't exist)/get Synapse Table to track team members
2. Get current list of team members
3. Compare current list with Synapse Table
4. email user when there are new members

Hi Sage-Bionetworks/synapsemonitor!

This is a one-off automatically generated pull request from LGTM.com :robot:. You might have heard that we’ve integrated LGTM’s underlying CodeQL analysis engine natively into GitHub. The result is GitHub code scanning!

With LGTM fully integrated into code scanning, we are focused on improving CodeQL within the native GitHub code scanning experience. In order to take advantage of current and future improvements to our analysis capabilities, we suggest you enable code scanning on your repository. Please take a look at our blog post for more information.

This pull request enables code scanning by adding an auto-generated codeql.yml workflow file for GitHub Actions to your repository — take a look! We tested it before opening this pull request, so all should be working :heavy_check_mark:. In fact, you might already have seen some alerts appear on this pull request!

Where needed and if possible, we’ve adjusted the configuration to the needs of your particular repository. But of course, you should feel free to tweak it further! Check this page for detailed documentation.

Questions? Check out the FAQ below!

FAQ

The scheduled job product in the Sage Service Catalog does not allow for a space in a parameters in the command line (see ticket). So the way the rate is specified needs to changed to remove any requirement for a space in a string.

Some alternatives:

1. Specify like 1h or 1d for 1 hour or 1 day
2. Two parameter specification of the rate --unit {hour,day} --value {#}

I have a preference for option 2 from a user standpoint as all of the information required will be within the command line help. For 1, the user would have to have some a priori knowledge of industry standards or have to read the documentation outside of the command line help.

Add auto documentation - will improve over time. This Github action does several steps whenever there is a push to the develop branch:

• It uses pdoc3 and creates the documentation
• Pushes the changes to thegh-pages branch
• The gh-pages branch has a index.html

Fixes #58

synapse login function now searches for synapse credentials in the following order:

1. SCHEDULED_JOB_SECRETS (+ json parsing for SYNAPSE_AUTH_TOKEN)
2. SYNAPSE_AUTH_TOKEN
3. config file path from function argument

Service Catalog scheduled jobs require Synapse PAT to be passed as a SECRET and extracted from an environmental variable in JSON format:

https://sagebionetworks.jira.com/wiki/spaces/SC/pages/938836322/Service+Catalog+Provisioning#Passing-Synapse-access-token-as-a-secret-using-environment-variables%3A

Will probably have to modify the synapse_login() function: https://github.com/Sage-Bionetworks/synapsemonitor/blob/ab78f60e1809a28221dd8d945c9b7a5c4bdc4a95/synapsemonitor/main.py#L136

Login should probably follow synapseclient default order of precedence for accessing credentials: https://python-docs.synapse.org/build/html/index.html#synapseclient.Synapse.login

1. Check first for SYNAPSE_AUTH_TOKEN
2. Check for SCHEDULED_JOB_SECRETS and parse to get SYNAPSE_AUTH_TOKEN value
3. Check for existence of .synapseConfig file from specified path in function argument

Bumps certifi from 2021.10.8 to 2022.12.7.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

I'm not sure if there is a functionality already, but I found if I create a fileview and only include folder in its scope. The view is not changing the modifiedOn value when a file is updated/uploaded to the scoped folders but it is changing when a subfolder is added. If we are using files in the view scope, the modifiedOn value reflects the most recent changes but would send us hundreds of SynIDs that would be less informative than only giving us the folders that have content changed.

Currently, synapsemonitor accepts a list of individual Synapse users to notify. The user should also be able to specify a team name in order to notify all members of the team that an entity has been modified.

Currently, the _traverse function is limited to collecting file, folder, and project entity types. Full list for getChildren includes ['folder', 'file', 'table', 'link', 'entityview', 'dockerrepo']. And there are probably other missing types here like 'datasets'.

Implementation would require a full mapping between the entity 'concreteType' and the getChildren 'includeTypes' strings.

'org.sagebionetworks.repo.model.FileEntity' --> 'file' 'org.sagebionetworks.repo.model.Folder' --> 'folder' 'org.sagebionetworks.repo.model.Project' --> 'project' 'org.sagebionetworks.repo.model.table.EntityView' --> 'entityview' 'org.sagebionetworks.repo.model.table.TableEntity' --> 'table' 'org.sagebionetworks.repo.model.Link' --> 'link' ? --> 'dockerrepo'