Resource Surveillance Roadmap
Introduction
Welcome to the page for the Surveilr Roadmap! Our goal with this part is to give our users an open picture of our ongoing work and planned features. In-depth details on our next weekly releases as well as our longer-term goals and concepts may be found here. Our intention is to keep you updated and involved as we work to add new features and capabilities to Surveilr. We’ll be updating this plan every week so you always have the most recent details regarding our development path.
Weekly Releases (Next 4 Weeks)
Week 1 (August 26 - 30, 2024)
- Non-Interactive Authentication for Microsoft 365 Integration
- Non-Interactive Authentication
- Automatic retrieval of access and refresh tokens.
- Secure cache for refresh tokens.
- Command:
surveilr ingest imap microsft-365 -i="<CLIENT_ID>" -s="<CLIENT SECRET>" -t="<TENANT_ID>"
- Interactive Authentication
- Require either
-m auth-code
or-m device-code
to initiate interactive authentication.
- Require either
- Orchestration Infrastructure
- CSV Transformation Bug
- Introduced a new argument
-table-name-patterns
(or shorthand-p
) to thesurveilr admin merge
command.
- End To End Tests
Week 2 (September 2 - September 6, 2024)
- Remote Automated Osquery Setup for
surveilr
- Implement a remote automated setup for Osquery within
surveilr
usingxxh
, enabling streamlined deployment and configuration.
- Implement a remote automated setup for Osquery within
- RSSD Schema Migration
- Utilize Atlas with
surveilr
SQL notebooks to create migrations for RSSDs, ensuring smooth transitions and updates.
- Utilize Atlas with
Week 3 (September 9 - September 13, 2024)
- Integrate DuckDB Library to Orchestrate Ingestion Center Functionality
- Develop a new
surveilr transform
command with options for--csv
,--excel
,--parquet
, and--all
. This command will process alluniform_resource
entries that match the specified file formats requested by the user. Currently,surveilr
performs some transformations like converting*.xml
to JSON and also*.csv
with thetransform csv
command, but this currently uses the inbuiltcsv
conversion inrusqlite
. The major aim of this newtransform tabular
feature will be to perform ETL/ELT to tables. - When the
--sql
flag is used, generate DuckDB SQL toSTDOUT
. When the--attach
flag is used, includeATTACH
to the RSSD. - Ensure all code functions correctly with an external DuckDB call. If the
--sql
flag is not used, generate DuckDB SQL and utilize the Rust crate to execute the DuckDB SQL viaATTACH
for the actual ETL/ELT process.
- Develop a new
Longer-term Projections (Next 3 Months)
October
Create an NPM Package for Installing surveilr
-
Objective: To streamline the installation process of
surveilr
in JavaScript projects by developing a custom NPM Package that integrates seamlessly with existing NPM workflows. -
Description: We aim to create a custom NPM Package designed to facilitate the installation and integration of
surveilr
into JavaScript projects. This package will simplify the setup process, ensuring that developers can easily incorporatesurveilr
into their existing workflows without encountering common installation hurdles. -
Key Features:
- Seamless Integration: Ensure the package integrates smoothly with existing NPM workflows.
- Simplified Installation: Streamline the setup process for developers incorporating
surveilr
. - Comprehensive Documentation: Provide detailed documentation to guide users through the installation and integration process.
-
Tasks:
-
Package Design:
- Define the package structure and dependencies.
- Plan for compatibility with various JavaScript frameworks and environments.
-
Development:
- Implement the core functionalities required for the
surveilr
installation. - Develop scripts to automate common setup tasks.
- Implement the core functionalities required for the
-
Testing:
- Perform thorough testing to ensure reliability and compatibility.
- Collect feedback from beta testers and make necessary adjustments.
-
Documentation:
- Create detailed documentation covering installation steps, usage instructions, and troubleshooting.
- Include example projects and code snippets to demonstrate usage.
-
Release:
- Prepare the package for release on the NPM registry.
- Announce the release and provide support channels for user feedback and issues.
-
Create a Maven Plugin for Installing surveilr
-
Objective: To simplify the installation process of
surveilr
in Maven projects by developing a Maven plugin that automatically handles all necessary dependencies and configurations. -
Description: We aim to create a Maven plugin designed to facilitate the installation and integration of
surveilr
into Maven projects. This plugin will streamline the setup process, ensuring that developers can easily incorporatesurveilr
into their existing workflows without encountering common installation hurdles. -
Key Features:
- Automatic Dependency Management: Handle all necessary dependencies required for
surveilr
integration. - Simplified Configuration: Automatically configure settings needed for
surveilr
within Maven projects. - Comprehensive Documentation: Provide detailed documentation to guide users through the installation and integration process.
- Automatic Dependency Management: Handle all necessary dependencies required for
-
Tasks
-
Plugin Design:
- Define the plugin structure and dependencies.
- Plan for compatibility with various Maven project setups.
-
Development:
- Implement the core functionalities required for
surveilr
installation. - Develop automation scripts for dependency management and configuration setup.
- Implement the core functionalities required for
-
Testing:
- Perform thorough testing to ensure reliability and compatibility.
- Collect feedback from beta testers and make necessary adjustments.
-
Documentation:
- Create detailed documentation covering installation steps, usage instructions, and troubleshooting.
- Include example projects and configuration snippets to demonstrate usage.
-
Release:
- Prepare the plugin for release on the Maven Central repository.
- Announce the release and provide support channels for user feedback and issues.
-
Windows winget
Installation Support
-
Create Manifest File
- Description: Develop a manifest file required by WinGet to facilitate the installation of
surveilr
on Windows machines. - Tasks:
- Define the metadata and configuration settings for the
surveilr
package. - Ensure the manifest file complies with WinGet specifications.
- Define the metadata and configuration settings for the
- Description: Develop a manifest file required by WinGet to facilitate the installation of
-
Add Uncompressed
surveilr.exe
to Release Assets- Description: Upload the uncompressed
surveilr.exe
executable to the release assets in the GitHub repository. - Tasks:
- Ensure the latest version of
surveilr.exe
is uncompressed and available in the release assets. - Verify the executable’s integrity and functionality.
- Ensure the latest version of
- Description: Upload the uncompressed
-
Add CI Action to Update Manifest File Upon New Release
- Description: Implement a Continuous Integration (CI) action to automatically update the manifest file whenever a new release of
surveilr
is published. - Tasks:
- Set up a CI workflow to trigger on new releases.
- Update the manifest file with the latest release information.
- Test the CI action to ensure it performs the update correctly.
- Description: Implement a Continuous Integration (CI) action to automatically update the manifest file whenever a new release of
Add gitSupplier feature to surveilr
-
Objective: To enhance
surveilr
’s capabilities by integrating the gitSupplier feature from udi-pgp-sqld, enabling it to monitor and analyze git commit logs from both local and remote repositories. -
Description: The gitSupplier feature will be integrated into
surveilr
, supporting two modes for ingesting git commit logs: Local Mode and Remote Mode. This integration will allowsurveilr
to monitor and analyze git commit logs from specified local git repository directory paths and remote GitHub repository URLs, providing a versatile tool for development and operations teams. -
Modes
- Local Mode:
- Ingests git commit logs from a specified local git repository directory path.
- Remote Mode:
- Ingests git commit logs from a specified remote GitHub repository URL.
- Local Mode:
-
Purpose: The integration of the gitSupplier feature will enhance
surveilr
’s capabilities by enabling it to monitor and analyze git commit logs from both local and remote repositories. This will provide a more versatile tool for development and operations teams, allowing for comprehensive source control surveillance. -
Expected Benefits:
- Enhanced Monitoring Capabilities: Ability to monitor git commit logs from both local and remote repositories.
- Increased Flexibility: More options for managing source control surveillance.
- Seamless Tracking: Ability to track changes and activity across different environments.
-
Tasks:
-
Feature Design:
- Define the integration plan for the gitSupplier feature.
- Ensure compatibility with existing
surveilr
architecture.
-
Development:
- Implement Local Mode to ingest git commit logs from a specified local directory path.
- Implement Remote Mode to ingest git commit logs from a specified GitHub repository URL.
-
Testing:
- Perform thorough testing for both modes to ensure reliability and accuracy.
- Collect feedback from beta testers and make necessary adjustments.
-
Documentation:
- Create detailed documentation covering configuration steps, usage instructions, and troubleshooting for both modes.
- Include examples and code snippets to demonstrate usage.
-
Release:
- Prepare the updated version of
surveilr
with the integrated gitSupplier feature for release. - Announce the release and provide support channels for user feedback and issues.
- Prepare the updated version of
-
Add Regex-Based Filtering for Attachment Extraction in surveilr
-
Objective: To enhance the flexibility and precision of attachment management in
surveilr
by introducing a new command-line argument,--extract-attachments-filter
, allowing users to selectively extract attachments based on a regular expression (regex). -
Description: Currently,
surveilr
handles attachments for each resource based on the--extract-attachments
flag, which accepts the values no, yes, oruniform-resource
. However, a blanket approach may not be suitable in all scenarios. To address this, we propose the introduction of the--extract-attachments-filter
argument, which will allow users to specify a regex to match the names of specific attachments they wish to extract. This enhancement will provide more granular control over attachment extraction, catering to various user needs and use cases. -
Key Features:
- Selective Extraction: Enable users to selectively extract attachments based on a regex provided through the
--extract-attachments-filter
argument. - Flexible Management: Allow more precise control over which attachments are extracted, improving the overall flexibility of the
surveilr
tool.
- Selective Extraction: Enable users to selectively extract attachments based on a regex provided through the
-
Tasks:
-
Feature Design:
- Define the syntax and usage of the
--extract-attachments-filter
argument. - Ensure compatibility with existing
--extract-attachments functionality.
- Define the syntax and usage of the
-
Development:
- Implement the
--extract-attachments-filter
argument. - Integrate regex processing to match specific attachment names for extraction.
- Implement the
-
Testing:
- Perform thorough testing to ensure the new argument works as expected.
- Validate regex matching and extraction accuracy.
- Collect feedback from beta testers and make necessary adjustments.
-
Documentation:
- Create detailed documentation covering the usage of the
--extract-attachments-filter
argument. - Provide examples and scenarios demonstrating the benefits of selective extraction.
- Create detailed documentation covering the usage of the
-
Release:
- Prepare the updated version of
surveilr
with the new--extract-attachments-filter
argument for release. - Announce the release and provide support channels for user feedback and issues.
- Prepare the updated version of
-