The Impact of Journal Weighting Scheme Characteristics on Research Output Measurement in Economics: The New Zealand Case

David L. Anderson, John Tressler

Abstract


Abstract: In this study we test for the power or aggressiveness of various journal weighting schemes, especially those based on the recursive adjustment methodology first developed by Liebowitz and Palmer. Using data generated by New Zealands academic economists, we provide quantitative measures of the differences between recursive adjustment-based schemes and selected alternatives. We then compare the performance of economics departments under each of our journal weighting schemes and, for comparison purposes, one based on direct citation counts. We find departmental rankings based on selected recursive adjustment schemes to be relatively stable, but these rankings differed substantially from those generated by our alternative schemes. This suggests that departmental hiring practices and research strategies must be sensitive to the type of funding scheme employed. In particular, research on domestic and regional issues is likely to be unattractive to researchers if a high-powered journal weighting scheme is adopted as the official standard since regional journals, the natural outlet for such work, are frequently zero-weighted by such schemes.


Keywords


Economics Departments, Research Output, Research Assessment Measures, Citations, Impact factors

Full Text:

Untitled () PDF

References


Anderson, D.L. and Tressler, J. (2008a). Research Output in New Zealand Economics Departments 2000-2006. New Zealand Economic Papers, 42(2), 155-189.

Anderson, D.L. and Tressler, J. (2008b). An Analysis of New Zealand Economists Research Output 2000-2006, Working Paper 20/08, Economics Department, University of Waikato, Hamilton, New Zealand.

Anderson, D.L. and Tressler, J. (2011). Ranking Economics Departments in Terms of Residual Productivity: New Zealand Economics Departments, 2000-2006. Australian Economic Papers, 50(4), 157-168.

Atkinson, A.B. (1970). On the Measurement of Inequality. Journal of Economic Theory, 2(3), 244-263.

Australian Research Council. (2010). Excellence in Research for Australia. Retrieved from www.arc.gov.au/era, 15 August 2010.

Baltagi, B.H. (2007). Worldwide Econometrics Rankings: 1989-2005. Econometric Theory, 23(5), 952-1012.

Chang, C.L., McAleer, M. and Oxley, L. (2011). What Makes a Great Journal Great in Economics? The Singer not the Song. Journal of Economic Surveys, 25(2), 326-361.

Coupe, T. (2003). Revealed Performances: Worldwide Rankings of Economists and Economics Departments, 1990-2000. Journal of the European Economic Association, 1(6), 1309-1345.

Chiu, W. H. (2007). Intersecting Lorenz Curves, the Degree of Downside Inequality Aversion, and Tax Reforms. Social Choice and Welfare, 28(3), 375-399.

Dusansky, R. and Vernon, C.J. (1998). Rankings of U.S. Economics Departments. Journal of Economic Perspectives, 12(1), 157-170.

Engemann, K. M. and Wall, H.J. (2009). A Journal Ranking for the Ambitious Economist. Federal Reserve Bank of St. Louis Review, 91(3), 127-139.

Garfield, E. (1972). Citation Analysis as a Tool in Journal Evaluation. Science, 178, 471-479.

Garfield, E. (2003). The Meaning of the Impact Factor. International Journal of Clinical and Health Psychology, 3(2), 363-369.

Gibson, J. (2000). Research Productivity in New Zealand University Economics Departments: Comments and Update. New Zealand Economic Papers, 34(1), 73-87.

Goldfinch, S. (2003). Investing in Excellence? The Performance-Based Research Fund and its Implications for Political Science Departments in New Zealand. Political Science, 55(1), 39-53.

Henrekson, M. and Waldenstrom, D. (2011). How Should Research Performance be Measured? A Study of Swedish Economists. Manchester School, 79(6), 1139-1156.

Hodder, A.P.W. and Hodder, C. (2010). Research Culture and New Zealands Performance-Based Research Fund: some insights from bibliographic compilations of research outputs. Scientometrics, 84(3), 887-901.

Kalaitzidakis, P., Mamuneas, T. and Stengos, T. (1999). European Economics: An Analysis Based on Publications in the Core Journals. European Economic Review, 43(4-6), 1150-1168.

Kalaitzidakis, P., Mamuneas, T., and Stengos, T. (2003). Rankings of Academic Journals and Institutions in Economics. Journal of the European Economic Association, 1(6), 1346-1366.

Kodrzycki, Y. K. and Yu, P. (2006). New Approaches to Ranking economics Journals. B.E. Journal of Economic Analysis and Policy: Contributions to Economic Analysis and Policy, 5(1), Article 24.

Laband, D. and Piette, M. (1994). The Relative Impact of Economics Journals. Journal of Economic Literature, 32(2), 640-666.

Liebowitz, S.J. and Palmer, J.P. (1984). Assessing the Relative Impact of Economics Journals. Journal of Economic Literature, 22(1), 77-88.

Macri, J. and Sinha, D. (2006). Rankings Methodology for International Comparisons of Institutions and Individuals: An Application to Economics in Australia and New Zealand. Journal of Economic Surveys, 20(1), 111-156.

Mason, P., Steagall, J. And Fabritius, M. (1997). Economics Journal Rankings by Type of School: Perceptions Versus Citations, Quarterly Journal of Business and Economics, 361), 69-79.

Millmow, A. (2011). Our Friends across the Ditch: A Brief Sketch of the New Zealand Academic Economic Profession. Economic Papers, 30(2), 273-278.

Moosa, I. (2011). The Demise of the ARC Journal Ranking Scheme: an Ex-Post Analysis of the Accounting and Finance Journals. Accounting and Finance, 51(3), 809-836.

Mukhopadhyay, T. And Sarkar, S. (2010). Rankings of Economics Journals and Departments in India. Working Paper 2010-021, Indira Gandhi Institute of Development Research, Mumbai, India.

Neri, F. and Rodgers, J.R. (2006) Ranking Australian Economics Departments by Research Productivity. Economic Record, 82(Suppl.), S74-S84.

OECD. (2010). Performance- based Funding for Public Research in Tertiary Education Institutions. Workshop Proceedings. OECD Publishing, Retrieved from http://dx.doi.org/10.1787/9789264094611-en.

Oswald, A. J. (2007). An Examination of the Reliability of Prestigious Scholarly Journals: Evidence and Implications for Decision-Makers. Economica, 74(293), 21-31.

Palacios-Huerta, I. and Volij, O. (2004). The Measurement of Intellectual Influence. Econometrica. 72(3): 963-977.

Ravallion, M. and A. Wagstaff (2011). On Measuring Scholarly Influence by Citations. Scientometrics, 88(1), 321-337.

Scott, L.C. and Mitias, P.M. (1996). Trends in Rankings of Economics Departments in the U.S.: An Update. Economic Inquiry, 34(2), 378-400.

Sinha, D., Macri, J., and McAleer, M. (2010). On the Robustness of Alternative Rankings Methodologies: Australian and New Zealand Economics Departments, 1988-2002. Applied Economics, 42(11), 1257-1268.

Starbuck, W.H. (2005). How much Better are the Most-Prestigious Journals? The Statistics of Academic Publication. Organization Science, 16(2), 180-200.

Tressler, J. and Anderson, D.L. (2012). Using Citations to Measure Research Output in New Zealand Economics Departments: The problem of Long and variable Lags. Agenda: A Journal of Policy Analysis and Reform, 19(1), 17-37.

Vanclay, J.K. (2011). An Evaluation of the Australian Research Councils Journal Ranking. Journal of Informetrics, 5(2), 265-274.

Wall, H. J. (2009). Dont Get Skewed Over by Journal Rankings. B.E. Journal of Economic Analysis and Policy: Topics, 9(1), Article 34.

Zimmermann, C. (2007). Academic Rankings with RePEc, Working Paper 2007-36, Department of Economics Working Paper Series, University of Connecticut, Storrs, Connecticut, U.S.A.




DOI: http://dx.doi.org/10.5202/rei.v3i3.95

Bookmark and Share


E-ISSN 2038-1379 -  2009-2023 University of Perugia