Available since version 4.1.0
This feature is not AEM as a Cloud Service compatible, and can only be used on AEM 6.5.
(RECOMMENDED FOR NON-PRODUCTION USE ONLY)
Remote Assets allows you to configure an AEM server to automatically sync in the assets from another, remote AEM server, on an as-needed basis. Remote Assets does not require anything to be installed on the remote AEM server.
Why do I need Remote Assets?
Pulling down a copy of the production site pages to a local or otherwise non-production server is simple. But, pulling down all of the assets associated with that site? Not so much. A simplistic, brute-force solution might be to use AEM’s package manager to bundle up the entire DAM, copy it down to your server, and call it a day, right? Not exactly. Copying the entire DAM can be problematic in terms of disk space, network transfer, and package size. How can this be done in a way that leaves behind assets that are not applicable to the site? And even if you can do it manually, how do you keep up to date when assets on the remote server change?
In short, you need a solution that achieves all of the following objectives:
- Access to all remote AEM Assets from another AEM Sites instance
- Remote AEM Assets nodes stored locally in AEM Sites for seamless authoring w/OOTB features
- Copies of just the required remote AEM Assets binary files, not binaries for the entire DAM
- All accomplished in an automated fashion
How does Remote Assets Work?
The mentality of Remote Assets is “sync what you need, when you need it.” To accomplish this goal, Remote Assets syncs assets from the remote system in a two-step process to optimize sync time and disk usage on the local server.
The first step for sync’ing assets is a “node sync,” where the full node trees for assets (and tags) configured by an admin or developer are sync’d to the local server. When the node sync takes place, the binary files of sync’d assets are replaced with placeholder binary files that are much smaller than typical “real” assets. This allows the entire set of assets to be sync’d to the local server at a fraction of the disk space and sync time needed if instead the entire set of assets were transferred as a content package. Having the entire node structure present on the local server allows assets to be searched and accessed as if they are fully present on the local server, supporting all OOTB functionalities.
The second step for sync’ing assets is a “binary sync,” which occurs on an individual asset basis. This sync is triggered whenever a non-admin user references a “remote” asset in any way. Here are just a few events that can trigger a binary sync:
- Author executes an asset search where a remote asset is in the search results
- Author opens a folder in the DAM, revealing remote assets within
- Author views remote assets via the asset finder on the page editor interface
- End user views a page where a remote asset is referenced
When any of these (or similar) actions take place by a non-admin user, the system detects that the requested asset is a “remote” asset that doesn’t yet have its binary files sync’d from the remote server, and in real time while the current request is processing, syncs the binary files into the local DAM “just in time.”
Once an asset has completed sync of its binary files to the local server, it is no longer considered a “remote” asset, and from that point forward there is no performance penalty in accessing the asset as it is now a “real” asset the same as if it were uploaded directly to the server. A “real” asset can once again become a “remote” asset during a subsequent node sync, but only if it has been updated on the remote server since the last node sync.
- Keeping Dev/QA/Stage servers and websites (author and/or publish) up to date with Production assets
- Dynamically pulling assets as-needed to localhost servers used for development or other business purposes
The Remote Assets feature should be considered a beta feature for non-production use only. It’s a very useful feature, but does come with some potential pitfalls that should be understood before using this feature. We look forward to future pull requests that help put more “guard rails” around this feature.
- DAM workflow launchers can be triggered by Remote Assets, resulting in poor performance and possibly even server crashes. This is avoided by following Step 1 in the “How to Use” section.
- There is no limit to how many assets a single request can trigger a binary sync for. Binary sync is, however, disabled for the admin user and for all service users to help avoid unintended syncs of large numbers of assets.
- There is no size limit on assets that can trigger a binary sync. If you have extremely large assets in a certain part of your DAM, you may wish to avoid sync’ing that part of the DAM.
- Sync of binaries is executed as a synchronous operation to provide the “magical” experience of assets being present on first request. In most cases, this results in the best user experience. However, a request involving remote assets may take significant time to complete if the connection between the local and remote server is slow, the requested assets are large, and/or the number of requested assets is very high. A high speed connection between the local and remote server will help mitigate this issue, and once remote asset binary files are sync’d for a set of assets there is no longer a delay in requesting those assets again.
- A server configured with remote assets will likely grow in size over time as more and more asset binary files are sync’d from the remote server. You may wish to manually purge assets from time to time.
- For use cases where the DAM is using a shared document store (e.g. Amazon S3) for the binaries, the benefits of Remote Assets would be limited. As such, Remote Assets does not currently support this use case.
PLEASE USE REMOTE ASSETS WITH CARE
How to Use
- (VERY IMPORTANT) Configure DAM Workflow Launchers to Ignore Remote Assets Updates
- Enable and Configure the Remote Assets Feature
- Manually Trigger or Schedule an Asset Node Sync
- Experience On-Demand Asset Binary Syncs
Configure DAM Workflow Launchers to Ignore Remote Assets Updates
By default, and on most AEM installations, there are numerous workflows that execute for creates and updates of assets
/content/dam path. Since Remote Assets will likely be sync’ing assets to this path, it is extremely
important that you configure these workflows to ignore activity from the Remote Assets feature, otherwise asset sync
activities may trigger numerous workflows. These unnecessary workflow executions can significantly slow the local
AEM server down, and in extreme situations can even crash the local AEM server.
Preventing workflow issues is as simple as updating all workflow launchers that execute on paths below
To do this…
- Navigate to AEM Workflow Launchers at http://localhost:4502/libs/cq/workflow/admin/console/content/launchers.html
- Open up Properties for each workflow with a path that starts with
- Ensure the
Exclude Listfield contains the value
- NOTE #1: Some launchers will already have other values inside
Exclude List. In those cases, be sure to add a
- NOTE #2: Some launchers will already have
event-user-data:changedByWorkflowProcessin place. For those launchers, no update is needed.
- NOTE #1: Some launchers will already have other values inside
Enable and Configure the Remote Assets Feature
Remote Assets requires two OSGi service configurations to be fully enabled.
The first required OSGi service configuration is simply a blank OSGi configuration file for the
RemoteAssetDecorator service, which is responsible for sync’ing a remote asset’s binary file the first time it is
requested by a non-admin user.
This is done via the Felix console or by deploying a blank OSGi configuration file from your project deployment.
The second required OSGi service configuration involves all of the main configuration values for the Remote Assets
feature, which are configured within the
- Fully qualified URL to the remote AEM server from which to sync assets
- Should be
server.insecureis set to
true(see applicable warnings)
- Username for logging into the remote AEM server
- This can be any user that has permissions to view assets and tags on the remote server - write permissions are not required
- Password for logging into the remote AEM server
- Whether or not to allow
server.urlto use the unsecure
httpprotocol, potentially exposing the authentication traffic with the remote AEM server
- It is recommended to only set
truefor traffic within a private network, where risk of exposing login credentials for the remote AEM server is reduced
- Array of JCR paths for which to recursively sync tags from the remote server to the local server
- Array of JCR paths for which to recursively sync assets from the remote server to the local server
- Number of minutes to wait to sync the binaries for a remote asset in the case of a failed attempt
- It is recommended to use a higher value in cases where the local server does not have regular network access to the remote server (e.g. local developer server that requires VPN to access the remote server)
- Number of asset nodes to sync before saving and refreshing the session during a node sync (default: 100)
- The lower the number, the longer the sync will take
- The event user data string that will be set during all JCR manipulations performed by remote assets (default: changedByWorkflowProcess)
- This is used in the
Exclude Listof workflow launchers that listen to DAM paths (e.g. DAM Update Assets) to avoid unnecessary asset workflow processing
- Array of service user names (default: empty) that are allowed to trigger remote asset binary syncs
- By default, service user activity never triggers an asset binary sync
Example Remote Assets configuration:
Manually Trigger or Schedule an Asset Node Sync
Once Remote Assets is configured, you’ll then need to execute a node sync. You can either execute the sync manually, or schedule the node sync to happen on a regular schedule.
Manual node sync
To execute a manual node sync, go to the JMX console at http://localhost:4502/system/console/jmx and open the Remote Asset Node Sync service.
Simply click the
syncAssetNodes() function and
Invoke it. Depending on how big the folders you have configured to
sync are, this process can take anywhere from a few seconds to a few hours. Assuming the return value of the function
null the sync has completed successfully.
Note that executing a manual node sync will not automatically trigger future node syncs, so you may wish to periodically execute the sync to keep up to date with new assets and tags.
Scheduled node sync
To schedule a node sync to happen on a regular basis, you can configure the
via the Felix console or by deploying an OSGi configuration file with a cron scheduler string from your project
Example config for scheduling a Remote Assets node sync once per week, on Tuesday nights starting at 8:00PM:
Experience On-Demand Asset Binary Syncs
Once you’ve completed configuration of Remote Assets and executed your first node sync you can now experience Remote
Assets by accessing assets in AEM as any user other than
admin. If Remote Assets is working correctly, and you’re
logged in as someone other than
admin, you should
never see an asset’s “temporary” binary (often stamped with the words “REMOTE ASSET”) but rather should see assets in
their “true” form.
Refer to the video demonstration of Remote Assets for what you should be experiencing with Remote Assets, and if you aren’t then refer to the Troubleshooting section below.
As this feature is meant for non-production use, and involves important activities to have full visibility of, it is
recommended that you set the log level of the
com.adobe.acs.commons.remoteassets package at all times to DEBUG.
This is a good first step in debugging any issues.
If you are unable to diagnose your problem with increased logging or the help of the scenarios outlined below, please feel free to reach out to the author of the original feature at firstname.lastname@example.org.
Slow node sync
If executing a node sync is going slower than 100+ assets per minute…
- Double check that you’ve configured all DAM asset workflows to ignore remote asset activities - see Step 1 in “How to Use”
- Verify that you have a fast network connection between the local and remote servers.
Asset sync is missing tags
If an asset node sync is not setting tags on the sync’d assets, ensure you’ve configured the appropriate tag paths to sync from the remote server. Tags must be present on the local server in order for the node sync to apply them to sync’d assets.
Asset binary files not sync’ing, assets showing as “REMOTE ASSET”
If you’ve executed a node sync successfully, but all assets in the DAM are showing up with the temporary binary files denoting “REMOTE ASSET,” this can be caused by a few different scenarios:
- You are logged in as the
adminuser. Because the
adminuser is used for some back-end process in AEM, Remote Assets does not sync asset binary files for the
adminuser. To resolve, log in as a non-admin user.
- Your browser has cached temporary binary file versions of these assets from viewing them as the
adminuser. This is a very easy “gotcha” and resolved with a simple browser cache clear.
RemoteAssetDecoratorservice is not configured. This service is configured with a simple blank OSGi config file, as covered in the “How to Use” section.
- Connection from the local server to the remote AEM server is failing. If a binary asset sync experiences a network failure, the asset will load with the temporary binary.
- A previous sync of the asset’s binary files failed, and the configured “Failure Retry Delay” has not yet expired. In this situation, ensure your local server is able to connect to the remote AEM server, and once the “Failure Retry Delay” has expired the binary sync should work on the next request for that asset.