Dataverses, Harvesting, Exporting, and OAI Sets

A standard dataverse holds uploaded studies and defined collections.

A harvesting dataverse culls data from other dataverses or Networks by using the OAI protocols. You cannot change the contents of a harvesting dataverse, but you can do any of the following:

You can create and manage the following aspects for all dataverses in the Network:

Create a New Dataverse

To create a standard dataverse:

  1. Navigate to the Network homepage.
  2. Click Network Options in the top-right corner, scroll to Dataverses, Harvesting, Exporting and OAI Sets, and then click Create a New Dataverse.
    You see the Add a New Dataverse page.
  3. Click the radio button next to the type of dataverse that you choose to create.
    A Scholar dataverse requires that you enter first and last name when you create the dataverse. Choose this option if you are an individual publishing your own studies.
    A Basic dataverse does not require first and last name values. Choose this option if you are publishing or collecting studies for groups, publications, or other communities.
  4. If you selected a Scholar dataverse type, then enter a First Name and Last Name for your dataverse.
    These values become the default dataverse homepage name.
  5. Enter an affiliation for the dataverse.
    For example, if this dataverse if for a Scholar, then enter the university or department in which the scholar works.
  6. Enter the full name for the dataverse homepage.
    You can change the default value for a Scholar dataverse, if you choose.
    This is the name that is displayed on the dataverse's homepage.
  7. Enter an alias for the dataverse.
    This is the user-friendly URL by which the dataverse is reached directly, and which can be used as a link to the dataverse from external web pages.
  8. For a Basic dataverse, use the Group Assignment pull-down list and choose the dataverse group in which to add this dataverse.
  9. Click Save Changes to keep your changes, or click Cancel to lose your changes.
    On the Network homepage Coming Soon tab, the dataverse is displayed.

Create a New Harvesting Dataverse

You can create two types of harvesting dataverse:

  • OAI Harvesting Dataverse - This type of dataverse harvests data from OAI servers.
    Refer to Open Archives Initiative for information about OAI services.
  • Nesstar Harvesting Dataverse - This type of dataverse harvests data from Nesstar servers.
    Refer to Nesstar, owned by Norwegian Social Science Data Services (NSD), for information about Nesstar services.

When a dataverse harvests from another service, the first harvesting is a full harvest. For all other harvesting, an update is performed.

See the following topics for detailed information about how to create these dataverses:

Create a New OAI Harvesting Dataverse

To create a harvesting dataverse:

  1. Navigate to the Network homepage, and then click Network Options.
    You see the [Your] Dataverse Network Admin Options page.
  2. Scroll to the Dataverses, Harvesting, Exporting and OAI Sets options, and then click Create a New Harvesting Dataverse.
    You see the Harvest Dataverse page.
  3. In the Harvest Type field, click the OAI Server radio button.
  4. Enter a Name for the dataverse, which is displayed on the Network homepage.
  5. Enter an Alias, which is the short name for the dataverse used to build the site's URL.
    This field is case sensitive.
  6. Enter the Server URL for the network server from which you choose to harvest data, and then click the Validate button.
    If the server is valid, additional Harvesting fields are displayed for further configuration. If the server is not valid, the error message Please enter a valid url is displayed. Correct the server name, and then click Validate again.
    An example valid server is the IQSS DVN OAI server, http://dvn.iq.harvard.edu/dvn/OAIHandler.
  7. If harvesting sets are configured for the OAI server that you entered, then use the Harvesting Set pull-down list to select a valid set option.
    The default is No Set (harvest all).
  8. Use the Harvesting Format pull-down list to select the XML format to use in harvesting from that server.
    Possible values are DC (Dublin Core) and DDI, depending on the server's configuration.
  9. Use the Handle Registration pull-down list to select whether to register with HANDLE.NET any harvested studies that have these identifiers on the server from which you harvest them.
  10. You can enable harvesting on a scheduled basis, to refresh the contents of a harvest dataverse and keep it current with the server from which you harvest.
    To enable scheduled harvesting:

    1. Click the check box in the Schedule Harvesting field.
    2. Use the Scheduled Harvesting Period pull-down list to select Harvest daily or Harvest weekly.
      If you do not choose an option, there is no schedule applied to the harvesting function.
    3. If you choose Harvest daily, in the Scheduled Harvesting Hour of Day field set the time of day at which to perform the harvest.
    4. If you choose to Harvest weekly, in the Scheduled Harvesting Day of Week also set the day on which to perform the harvest.
  11. You can set permissions for all studies and files in the harvested dataverse to either Public or Restricted.
    To enable file permissions:

    1. Use the File Permissions pull-down list to make all study files in the harvested dataverse either Public or Restricted.
    2. If you choose to make files Restricted, use the Subset Permissions pull-down list to make subsetting of studies in the harvested dataverse either Public or Restricted.
    3. To enable specific users access to Restricted studies, enter a username in the field Allowed Users, if Files are Restricted, and then click the Add button.
      To remove access from a user, click the Remove User button beside that user's name in the Allowed Users field.
    4. To enable specific user groups access to Retricted studies, enter a group name in the field Allowed Groups, if Files are Restricted, and then click Add button.
      To remove access from a group, click the Remove Group button beside that group's name in the Allowed Groups field.
  12. Click Save or Cancel.
    If your save succeeds, the message Update Successful! is displayed.
    Click Network Options to return to the Network Admin Options page.

Note: Remember to release your harvesting dataverse to the public. See the Dataverse Admins Guide for information about releasing a dataverse.

Create a New Nesstar Harvesting Dataverse

To create a dataverse that harvests data from a Nesstar server:

  1. Navigate to the Network homepage, and then click Network Options.
    You see the [Your] Dataverse Network Admin Options page.
  2. Scroll to the Dataverses, Harvesting, Exporting and OAI Sets options, and then click Create a New Harvesting Dataverse.
    You see the Harvest Dataverse page.
  3. In the Harvest Type field, click the radio button for the Nesstar Server option.
  4. Enter a Name for the dataverse, which is displayed on the Network homepage.
  5. Enter an Alias, which is the short name for the dataverse used to build the site's URL.
    This field is case sensitive.
  6. Enter the Server URL for the Nesstar network server from which you choose to harvest data, and then click the Validate button.
    If the server is valid, additional Harvesting fields are displayed for further configuration. If the server is not valid, the error message Please enter a valid url is displayed. Correct the server name, and then click Validate again.
  7. Use the Harvesting Format pull-down list to select the XML format to use in harvesting from that server.
    The value DDI is available at this time.
  8. Use the Handle Registration pull-down list to select whether to register with HANDLE.NET any harvested studies that have these identifiers on the server from which you harvest them.
  9. You can set permissions for all studies and files in the harvested dataverse to either Public or Restricted.
    To enable file permissions:

    1. Use the File Permissions pull-down list to make all study files in the harvested dataverse either Public or Restricted.
    2. If you choose to make files Restricted, use the Subset Permissions pull-down list to make subsetting of studies in the harvested dataverse either Public or Restricted.
    3. To enable specific users access to Restricted studies, enter a username in the field Allowed Users, if Files are Restricted, and then click the Add button.
      To remove access from a user, click the Remove User button beside that user's name in the Allowed Users field.
    4. To enable specific user groups access to Retricted studies, enter a group name in the field Allowed Groups, if Files are Restricted, and then click Add button.
      To remove access from a group, click the Remove Group button beside that group's name in the Allowed Groups field.
  10. Click Save or Cancel.
    If your save succeeds, the message Update Successful! is displayed.
    Click Network Options to return to the Network Admin Options page.

Note: Remember to release your harvesting dataverse to the public. See the Dataverse Admins Guide for information about releasing a dataverse.

Manage Dataverses

You can edit and remove both standard and harvesting dataverses.

For harvesting dataverses, you can:

  • Define harvesting schedule.
  • Schedule harvesting.
  • Run harvester manually.

See the following topics for detailed information:

Edit Dataverse Setup

To edit options for a dataverse:

  1. Navigate to the Network homepage.
  2. Click Network Options in the top-right corner, scroll to Dataverses, Harvesting, Exporting and OAI Sets, and then click Manage Dataverses.
    You see the Manage Dataverses page.
  3. Click the name of the dataverse for which you choose to manage options.
    You see the My Options page for the dataverse that you selected.
    For information about the setup of standard dataverses, see Create a New Dataverse.
    For information about the setup of harvesting dataverses, see Create a New Harvesting Dataverse.

Define Harvesting Schedule

Use the Status column in the Harvest Dataverse list to access harvesting schedules as follows:

  1. Navigate to the Network homepage, and click Network Options.
    You see the [Your] Dataverse Network Admin Options page.
  2. Scroll to the Dataverses, Harvesting, Exporting, and OAI Sets options, and then click Manage Dataverses.
    You see the Manage Dataverses page.
  3. If no schedule is set up, the message Harvesting Schedule Not Defined is displayed in the row for that dataverse.
    To set up a harvesting schedule, click the Define Harvesting Schedule link.
    You see the Harvest Dataverse page, with values for the current configuration.
  4. Click the Scheduled Harvesting? check box to enable the schedule.
  5. Set up the following fields to enable the schedule. See Create a New Harvesting Dataverse for information about these fields:
    • Scheduled Harvesting Period - Select Harvest daily or Harvest weekly.
    • Scheudled Harvesting Hour of Day - Set the time of day at which to perform the harvest.
    • Scheduled Harvesting Day of Week - For a weekly harvest, also set the day on which to perform the harvest.
  6. Click Save to save the schedule, or click Cancel to loose the setup.

Schedule Harvesting

Use the Status column in the Harvest Dataverse list to enable or disable scheduled harvesting as follows:

  1. Navigate to the Network homepage, and click Network Options.
    You see the [Your] Dataverse Network Admin Options page.
  2. Scroll to the Dataverses, Harvesting, Exporting, and OAI Sets options, and then click Manage Dataverses.
    You see the Manage Dataverses page.
  3. If a schedule is set up but is not active, the message Not Scheduled is displayed in the row for that dataverse. A Schedule Haravesting button appears beside the message.
    If a schedule is set up and is active, the message Scheduled is displayed, and an Unschedule Haravesting button appears.
    To enable harvesting on the configured schedule, click the Schedule Harvesting button.
    To disable harvesting on the configured scheduled, click the Unshcedule Harvesting button.

Run Harvester Manually

To harvest a dataverse manually:

  1. Navigate to the Network homepage, and click Network Options.
    You see the [Your] Dataverse Network Admin Options page.
  2. Scroll to the Dataverses, Harvesting, Exporting, and OAI Sets options, and then click Manage Dataverses.
    You see the Manage Dataverses page.
  3. Click the Run Harvester Now button in the row corresponding to the dataverse you choose to harvest.
    The page changes to display Harvesting Currently Running in place of the Run Harvester Now button. All options for that dataverse are disabled until the harvesting activity is complete, or until you toggle the Schedule/Unschedule Harvesting button.

Remove Dataverses

To remove a dataverse from the Network:

  1. Navigate to the Network homepage.
  2. Click Network Options in the top-right corner, scroll to Dataverses, Harvesting, Exporting and OAI Sets, and then click Manage Dataverses.
    You see the Manage Dataverses page.
  3. Locate the entry for the dataverse that you choose to remove, and then click the Remove link in that row of the list.
    You see the Deleting a Dataverse page.
  4. To confirm the remove action, click the Delete button. Or click the Cancel button to keep the dataverse in the Network.
    If your delete was successful, you see a Delete Successful page.

Manage Dataverse Groups

You can organize dataverses into groups, to structure the Network homepage and help users to find a dataverse more easily. Create groups of dataverses that have common entities, such as subjects, authors, or dates. This is similar to creating collections within a dataverse. On the Network homepage, a dataverse group name is displayed in a rectangular gray box, and the dataverses in the group are arranged alphabetically under the name.

When you create the first new dataverse group in the Network, then all dataverses become members of a default dataverse group called Other. You can add dataverses to new groups, which moves them from the Other group. A dataverse can belong to more than one group, but use that organization only if it is useful.

You can do the following with dataverse groups:

Add Dataverse Group

To create a dataverse group:

  1. Navigate to the Network homepage.
  2. Click Network Options in the top-right corner, scroll to Dataverses, Harvesting, Exporting and OAI Sets, and then click Manage Dataverse Groups.
    You see the Dataverse Groups page.
  3. Click the Add Group link at the botton of the group list.
    You see the Dataverse Group page.
  4. Enter a Name for the group.
  5. Enter a Description for the group.
  6. To add a dataverse to this group, click the name of a dataverse that you choose to add in the All Dataverses box, and then click Add. Hold down the Ctrl key to select more than one dataverse, and then click Add to add them all to the group. Or, double-click the dataverse name in the All Dataverses box.
    You see the dataverses that you added in the Dataverses in this Group box.
    To remove dataverses from this group, click the dataverse name in the Dataverses in this Group box, and then click Remove. Hold down the Ctrl key to select more than one dataverse at a time to remove from them from the group. Or, double-click the name in the Dataverses in this Group box.
  7. When the dataverse group is set up, click the Save button to save your changes.
    You see the Dataverse Groups page, and see a message that your update was successful. On the Network homepage, the group name is displayed in a rectangular gray box, and the dataverses in the group are arranged alphabetically under the name.

Edit Dataverse Group Setup

To edit the setup or name of a dataverse group:

  1. Navigate to the Network homepage.
  2. Click Network Options in the top-right corner, scroll to Dataverses, Harvesting, Exporting and OAI Sets, and then click Manage Dataverse Groups.
    You see the Dataverse Groups page.
  3. In the list of groups, click the name of the group that you choose to edit.
    You see the Dataverse Group page.
  4. Edit the name or description of the group, or change the members.
    To add a new dataverse to the group, click the dataverse name in the All Dataverses field, and then click Add. To remove a dataverse from the group, click the dataverse name in the Dataverses in this Group field, and then click Remove.
  5. Click the Save button to save your changes.
  6. Navigate to the homepage.
    The group changes to reflect your new setup.

Change Dataverse Group Order

When you change the order in which dataverse groups are displayed on the Network homepage, you edit the order one time and save your changes. If you try to edit the order a second time without navigating through the Manage Dataverse Groups menu option, your changes have no effect.

To change the order in which dataverse groups are displayed on the Network homepage:

  1. Navigate to the Network homepage.
  2. Click Network Options in the top-right corner, scroll to Dataverses, Harvesting, Exporting and OAI Sets, and then click Manage Dataverse Groups.
    You see the Dataverse Groups page.
  3. In the list of groups, change the Display Order value higher to move the group lower on the homepage. Change the value lower to move the group higher on the homepage.
    If you change a value in the middle of the list, the order values after that entry change automatically to follow the new value sequentially.
  4. Click the Save button to save your changes.
  5. Navigate to the homepage.
    The order of groups listed on the Dataverse Groups page changes to reflect your new order.

Change Number of DVs Displayed per Group Page

Within a dataverse group, the member dataverses are displayed in four columns. You can modify the number of dataverses in a group that are displayed on a page. You also can define for all groups a default number of dataverses to display on one page.

If you set the Display Number to a value less than the number of dataverses in that group, Next and Previous links are added to the listing to support paging through the contents of the group.

To edit the number of dataverses that are displayed within a dataverse group page:

  1. Navigate to the Network homepage and click the Network Options link.
    You see the Network Options page.
  2. Under Dataverses, Harvesting, Exporting and OAI Sets, click Manage Dataverse Groups.
    You see the Dataverse Groups page.
  3. To set the default number of dataverses to display per page for all groups, edit the standalone field contained in the text of item 1.
    This field is preceeded by the text Number of dataverses to display per group, per page.
  4. To modify the default value of dataverses to display per page for a specific group, edit the Display Number field for that group's entry.
    The individual group's Display Number setting overrides for that group only the default value.
  5. Click Save to save your changes.
    The page setup for each group takes effect on the Network homepage.

Delete Dataverse Group

If you delete a dataverse group, any dataverse that is a member of that group and is not a member of another group is moved to the default Other group.

To delete a dataverse group:

  1. Navigate to the Network homepage.
  2. Click Network Options in the top-right corner, scroll to Dataverses, Harvesting, Exporting and OAI Sets, and then click Manage Dataverse Groups.
    You see the Dataverse Groups page.
  3. In the list of group, click the Delete check box in the row of the group that you choose to delete.
  4. Click the Save button to save your changes.
    The entry for that dataverse group is removed from the list of groups, and the member dataverses return to the default group, Other.

Schedule Study Exports

You can schedule how often studies in your Network are exported into XML that uses DDI (study and variable metadata) and Dublin Core (study metadata only) formats. These export files are used by the Network's OAI server and for the preservation and sharing of your data. You also can edit an existing export schedule.

To set up an OAI export schedule for studies in your Network:

  1. Navigate to the Network homepage, and then click Network Options.
    You see the [Your] Dataverse Network Admin Options page.
  2. Click Schedule Study Exports.
    You see the Study Export Schedule page.
  3. Use the Export Period pull-down list to select whether to export your Network's metatdata on a daily or weekly basis.
  4. In the Export Hour of Day field, set the time of day at which to perform the export, based on a 24-hour clock.
    Enter an integer between 0 (midnight) and 23 (11 PM).
  5. In the Export Day of Week field, set the day on which to perform the export. Enter an integer between 1 (Sunday) through 7 (Saturday).
  6. Click Save to save the schedule, or click Cancel to lose your entries.
    If your save succeeds, you see the message Update Successful!
    Click Network Options to return to the Network Admin Options page.

To edit an existing export schedule, navigate to the Schedule Study Export option and modify the current configuration on that page.

Manage OAI Harvesting Sets

You can create a new set to be used by the OAI server that comes with your Network. Another DVN, or any other remote site, then can harvest the sets that you define here. If you do not define sets to be harvested, the default OAI call (http://localhost/dvn/OAIHandler) harvests the contents of your entire Network.

Read these topics for detailed information about how to manage OAI harvesting sets:

Create a New OAI Set

To create a set for harvesting of your Network's studies from other OAI clients:

  1. Navigate to the Network homepage, and then click Network Options.
    You see the [Your] Dataverse Network Admin Options page.
  2. Click Manage OAI Harvesting Sets.
    You see the OAI Sets defined in [Your] Dataverse Network page.
  3. Click the Create a New Set link at the top of the set list.
    You see the Edit OAI Set page.
  4. Enter a Set Name to be displayed when a harvesting OAI client calls your Network's OAI server to harvest from your Network.
  5. Enter a Set Spec value to use as a unique Identifier for this set.
    This value is the token passed from OAI server to client during harvesting.
  6. Enter a Query Definition from which to create the set.
    This query uses the same syntax as query collections, and is based on the Lucene search engine. See List of Metadata for information about query terms. Refer to the Documentation page for the latest version at the Lucene website and look for Query Syntax for full details.
  7. If you choose, enter a description about the OAI set to help define its contents.
  8. Click Save to save your settings, or click Cancel to lose them.
    You see the OAI Sets defined in [Your] Dataverse Network page, with your set listed.

Edit an OAI Set

To edit an existing OAI set:

  1. Navigate to the Network homepage, and then click Network Options.
    You see the [Your] Dataverse Network Admin Options page.
  2. Click Manage OAI Harvesting Sets.
    You see the OAI Sets defined in [Your] Dataverse Network page.
  3. Click the set name in the list for the set that you choose to edit.
    You see the Edit OAI Set page, containing the current set definition.
  4. Edit the values to change the set definition.
  5. Click Save to save your changes.
    You see the OAI Sets defined in [Your] Dataverse Network page again.

Delete an OAI Set

You can delete a set from your Network export at any time.
Note: There is no confirmation step when you delete an OAI set. Be sure that you choose the correct set to delete before you start this action.

To delete an OAI set from your Network export:

  1. Navigate to the Network homepage, and then click Network Options.
    You see the [Your] Dataverse Network Admin Options page.
  2. Click Manage OAI Harvesting Sets.
    You see the OAI Sets defined in [Your] Dataverse Network page.
  3. Click the Remove link in the row of the set that you choose to delete.
    Note: There is no confirmation step when you delete an OAI set.
    The selected set is removed from your Network export.